A recent strategy to circumvent the exploding and vanishing gradient problem in RNNs, and to allow the stable propagation of signals over long time scales, is to constrain recurrent connectivity matrices to be orthogonal or unitary. This ensures eigenvalues with unit norm and thus stable dynamics and training. However this comes at the cost of reduced expressivity due to the limited variety of orthogonal transformations. We propose a novel connectivity structure based on the Schur decomposition and a splitting of the Schur form into normal and non-normal parts. This allows to parametrize matrices with unit-norm eigenspectra without orthogonality constraints on eigenbases. The resulting architecture ensures access to a larger space of spectrally constrained matrices, of which orthogonal matrices are a subset. This crucial difference retains the stability advantages and training speed of orthogonal RNNs while enhancing expressivity, especially on tasks that require computations over ongoing input sequences.

This paper proposes nnRNN, which is the non-normal matrix relaxation to improve the expressivity of RNNs with unit eigenvalue constraints. Previous studies use the unitary or orthogonal matrix constraints to prevent the exploding and vanishing gradient problems in RNNs, but these constraints deteriorate the performance in empirical tasks due to the strong constraints. This paper relaxes the unitary matrix (which is a normal matrix that has eigenvalues on the unit circle) constraints as allowing the non-normal matrices that have the eigenvalues on a unit circle. In addition, this paper theoretically and empirically shows how the non-normal relaxation affects the dynamics of RNNs. This paper is well written, and the motivation of non-normality is explained thoroughly.

The paper contributes in a significant way to show how RNN able to deal with vanishing/exploding gradients can be obtained not only by enforcing recurrent weight matrices to be normal or orthogonal, but also by exploiting a larger set of matrices obtained by Schur matrix decomposition. Although the proposed contribution cannot be readily extended to gated units and the computational effort is increased because of the Schur matrix decomposition, the paper provides new insights that can be the basis for improved algorithms. The rebuttal helped to solve most of the issues raised by the reviewers.

A recent strategy to circumvent the exploding and vanishing gradient problem in RNNs, and to allow the stable propagation of signals over long time scales, is to constrain recurrent connectivity matrices to be orthogonal or unitary. This ensures eigenvalues with unit norm and thus stable dynamics and training. However this comes at the cost of reduced expressivity due to the limited variety of orthogonal transformations. We propose a novel connectivity structure based on the Schur decomposition and a splitting of the Schur form into normal and non-normal parts. This allows to parametrize matrices with unit-norm eigenspectra without orthogonality constraints on eigenbases. The resulting architecture ensures access to a larger space of spectrally constrained matrices, of which orthogonal matrices are a subset.

A recent strategy to circumvent the exploding and vanishing gradient problem in RNNs, and to allow the stable propagation of signals over long time scales, is to constrain recurrent connectivity matrices to be orthogonal or unitary. This ensures eigenvalues with unit norm and thus stable dynamics and training. However this comes at the cost of reduced expressivity due to the limited variety of orthogonal transformations. We propose a novel connectivity structure based on the Schur decomposition and a splitting of the Schur form into normal and non-normal parts. This allows to parametrize matrices with unit-norm eigenspectra without orthogonality constraints on eigenbases.